In this article, we explore some practical uses of AI driven automated text generation. We demonstrate how technologies like GPT-3, and more recently ChatGPT, can be used to better your business applications by automatically generating training and test data which can be used to bootstrap your machine learning models and validate them. We also illustrate some example uses of language transformations like transforming english into legalese or spoken text into written.

This article is a follow up to our recent article: Modern AI Text Generation, where we discussed state-of-the-art AI models for generating human realistic text. We extend that article by applying Open AI's GPT-3 model to a variety of practical business scenarios. For your convenience, we have made public the code needed to replicate our efforts along with installation scripts to get you quickly on your way to developing your own solutions via a Jupyter notebook. The source can be downloaded here. Please follow the instructions in the README file for installation.

efficient bootstrapping of text models

Generally speaking, large volumes of text are needed to solve some of the more complex problems tackled by deep learning models. As a testament, OpenAI essentially ingested the world's worth of communicated text on the internet to build its GPT-3 model, which is used to help solve a wide variety of NLP problems. However, there is often a practical limitation data scientists face:

Procuring large amounts of text data needed to train deep learning NLP models can be problematic or, in some cases, impossible.

There are several reasons for lack of data:

The data is publicly available, say on social media platforms where an API is provided, but the rate at which you can access the data is throttled.
The type of data you require is rare or non-existent in the public domain.
The data you require exists but sits behind an expensive pay wall or is proprietary.

Throttled APIs can be particularly frustrating given that ingesting sufficiently large datasets may take weeks or even months to procure. Perhaps even more frustrating is the fact that you may be limited in how well you can target the type of text you wish to procure and therefore need to spend lots of time combing through records, once you get them, to make sure that the quality is up to the standards you desire.

One workaround to this dilemma is to directly simulate the data that you may need to bootstrap your text learning model. We've seen how well GPT-3 can perform given prompt text that pushes the completions in the right direction, giving you the user quite a bit of control over the quality of the responses. We at Xyonix have used this approach before and it particularly served us well in cases where the text we sought was rare. We extend this approach to several possible business scenarios below for illustration of the approach. For each, we hypothesize the purpose of the company to help make the concept more realistic. We state up front that these are real completions created by GPT-3, mostly unaltered: in some scenarios we removed/altered punctuation or abbreviated completions to meld better with the bulk of the examples returned. In all cases, the level of realism is very high. So much so that it is easy to forget that these completions were made by a machine and not a human! Keep that in mind as you peruse the results and, as you do so, imagine extensions that you could create to better the data for your particular business application.

In the scenarios presented below, the GPT-3 model input and output text is highlighted as follows for convenient reading:

bolded text: this is the input text "prompt" to the model. In our scenarios, the prompts are the beginning of a sentence or paragraph we wish to complete or a set of examples serving as patterns that we wish for GPT-3 to extend.
italicized text: the output of the GPT-3 model, otherwise known as a "completion".

For example, in the text "Tom Cruise is a great actor and a great guy.", "Tom Cruise is" is the input prompt and "a great actor and a great guy." is the completion that GPT-3 has output; a real-world example by the way!

application: simulating restaurant reviews

You are the owner of several Lebanese restaurants and wish to quantify customer happiness based on anonymous feedback left by patrons. You plan on developing a sentiment analysis model to categorize your customers as either happy, unhappy, or neutral. In order to train your algorithm, you'd like to gather comments about Lebanese food specifically so that you might better understand why customers might be happy or unhappy with your food. You find that such data is simply unavailable or sparse online and so you turn to GPT-3 to help simulate customer experience response:

The Kibbeh tasted very similar to the ones I’ve had in Damascus. The yogurt sauce on top was quite thick and creamy. We also ordered a plate of Falafel. The Falafel were crunchy and not too oily. We finished the meal with a piece of Baklawa and a glass of Tamarind Mint Tea. The Baklawa was sweet and crunchy, but not very flavorful. I enjoyed the tamarind tea, but it was served lukewarm.
The chicken shawarma was okay but the chicken was a bit dry. Plus it was served on a flatbread which I didn’t really like. The sides were also a bit bland. The garlic sauce that was served with the chicken had a weird taste to it. The falafel wrap had no taste at all. I really didn’t like the falafel. It was dry and tasteless.
The falafel was really good and we came home stuffed from all of the hummus.
The baklava was very good. It was the first time I had tried it and I enjoyed it very much.
The Kanafeh I had was beyond delicious. I want to reminisce about this place for a day or two and then go back to the same. It was full of flavor from within, so nicely done...the Al Gahba nan Qatar is divine and portions are great.
The Kanafeh was not bad but it was not lip smacking. It is good to experiment and try new things. i would prefer Lebanese tea (with milk) over this, its better with a hibiscus sprig :)

Note that these were the first results returned from the API and are not cherry picked results. With very little effort then, you can procure realistic and interesting reviews specific to Lebanese food using GPT-3. With the mere mention of a specific type of food you are likely to serve mentioned in the prompt, GPT-3 was able to take that and run it to generate positive, neutral, and negative feedback on Lebanese food. In addition, there are a few misspelled words, Baklawa for example, which should be spelled Baklava, but it is important to include such misspellings because your customers are also likely to misspell a word or two, if not for their lack of experience in eating Lebanese food. This serves as a simple but effective example of the power of GPT-3 and how you could use it to very quickly bootstrap a sentiment analysis model for a Lebanese food restaurant.

application: video interview assessment

Your company specializes in assessing the suitability of job candidates for educational positions. You are looking to develop models that rank the quality of the candidates based on their responses to canned questions they are asked to address during an automated and recorded video interview. To bootstrap your models, however, would require realistic responses from typical candidates and currently you don't have access to thousands of volunteers to fake video interviews. You turn to GPT-3 to simulate responses from standard questions candidates will be prompted to answer. For example, one of the questions involves the reasons behind the teacher's interest in the posted position. Here are several responses GPT-3 generated based on the prompt in bold:

I am interested in the elementary teacher position because

it fits with my idea of "teaching". I believe that the most important factor involved in education is a student's interest in what is being taught. As such, I am most qualified to teach language arts and literature, history, and politics because I enjoy learning about those subjects.
I have a bachelor's degree in elementary education and I have a lot of experience teaching various grades. My favorite grade to teach is the primary grades.
I believe a properly trained and well-prepared teacher, along with supporting educational-materials will assist children in learning important foundational skills required to flourish. I know that my role in life directly influences the lives of children and I look forward to being a very positive influence in the lives of a classroom of students. I would confidently say that I am an extremely strong applicant for this position.
I am looking for a position that will allow me to be in the classroom with children and encourage them to learn and grow. I have experience working with children as well as tutoring them in the areas of reading, writing, and math. I am also a certified EMT and I have experience with children with special needs.
I enjoy working with children, especially in a challenging environment like a school. I have always been a hard worker and a quick learner. I would be an excellent teacher because I am friendly and easy to get along with, and I love working with children. I can relate to them and feel I can relate to their problems and concerns.

In these simulated examples, we see a wide spectrum of responses where our faux candidates have made statements about their relevant experience, philosophy of education, level of confidence, learning rates, and specialty areas. All of these attributes could be harvested in a model that ranked the suitability of the applicants to the offered position.

application: post treatment doctor survey

Your company specializes in developing surveys for healthcare. Hospitals are using your surveys to perform exit interviews with doctors and would like for you to divide those responses into meaningful subpopulations so that they may better serve their future patient population. This is a new and exciting extension of your survey product but you currently lack relevant responses to dial in your algorithms to divide the population into relevant personas. You can turn to GPT-3 to simulate responses with basic prompts that you expect to be on the questionnaire. Below, the prompts are in bold and the bulleted items are simulated responses of GPT-3.

I wish my doctor had

told me about the risks of Paxil. I was given this medication for anxiety, depression and an eating disorder. I have also taken other antidepressants in the past but I had horrible side effects.
told me that I'm not the only one. I wish he had told me the truth about the dangers of the device. The risks are clear and every woman having a coil fitted must be told what they are before they agree to it. This is not about guilt, it's about accountability.
told me that I didn’t need to take birth control to control my period and reduce cramps.
asked me whether I was a runner. I assume it would have been helpful information to him when he was dispensing advice. If he had asked, I could have told him that I ran the equivalent of a half marathon every day during my pregnancy. That is not an exaggeration.

The visit with my doctor was

very informative. She is very knowledgeable and helpful.
about the same as my last visit. She asked how I was doing, I told her I was doing good. She then proceeded to tell me that I was doing great and that she was glad to see me doing so well. She then went on to give me a list of things I could do to help with my recovery. She told me that I needed to continue to work on my walking.
good, I got to see my baby today, and I am due in July. I am so happy and excited. I went through a lot of stuff to get pregnant with my first child and I could not be happier.
good. He was very nice. He asked me a lot of questions and told me that he really liked my answers. He told me he has some diet suggestions for me and that he will be ordering another MRI of my brain. I am scheduled to see him again in 3 months.
the best I had ever had. He was very professional, yet friendly and easy to talk to. He listened to everything I said and did not rush me. He also explained everything that he was doing and I felt very comfortable with him. I would highly recommend him to anyone who is looking for a doctor.

application: assessing mental health

With the COVID-19 pandemic, threats of global warming, political unrest, and being isolated from friends at school, kids and teens have a lot on their minds lately and many are suffering from mental health as a result. Your company provides depression and threat assessment software for schools to use (with permission) to monitor messages exchanged on school computers and communication platforms. You wish to train a model to categorize at risk kids based in part by their communications. For lack of initial data, you decide to bootstrap your models with simulated text communication from school kids with relevant prompts. Here are some real GPT-3 completions that could be used to bootstrap your mental health models, given the prompt shown in bold.

I'm feeling down at school because

I missed so much work, and it's affecting my grades.
I'm sad about Mr. Pasimirvichevich coming back. I need help with school.
I don't have a boyfriend. I like a lot of the boys at school but none of them are my friends, they just flirt with me.
my little sister died last night.
I can't ride the bike anymore! I know our school is a total drag but it's still boring!
I used to love going there and didn't this morning.
my mum and dad aren't talking to each other. My sister and I can hear them shouting at each other in the rec room."
I had to wear a dress yesterday. It's very uncomfortable and I'm self-conscious. My mother is embarrassed about my haircut.
my friend is going to be gone a couple weeks for her dad's funeral. I don't understand why they have to leave there as soon as it starts, when we've still got two months of school until the end of summer...I'm not feeling so confident about my work on this book project, either…

These completions seem realistic and cover a wide range of topics including self-conscious behavior, family in-fighting, feelings of boredom, dating issues, and even very sad situations where a member of the family has passed away. Certainly, these are all experiences that we can relate to in going to school at a young age and that level of realism will help you kick start your models, giving them data in a wide variety of topics relevant mental health topics.

application: LEGAL TRANSLATION

Unlike previous examples where we discuss a hypothetical business and associated application, here we show off GPT-3's direct ability to perform language transformation. Specifically, we focus on examples of translation discourse, where stating things one way is translated by GPT-3 to be expressed another way. For example, you can use GPT-3 to transform plain language into legalese or vice versa. There are several reasons reasons for legal translation including:

Spot legal risks in non-legal documents or society at large;
Find legal arguments in legal documents like legislation or case law;
Construct and draft legal arguments like a contract or judgement;
Explain legal arguments to a client or a judge.

Our legal translation prompts were taken from this Tweet and the results that we got were impressive given that we only needed to provide four paired examples in the prompt for GPT-3 to pick up on the pattern. Here are some prompt endpoints and the resulting completions:

PLAIN ENGLISH: I am going to sue him in court if he fails to pay me my rent money that is 6 months past due.

LEGALESE: If the Company fails to pay rent for more than 3 months, then the landlord may terminate this agreement.

PLAIN ENGLISH: I have the right to seek employment elsewhere as I did not sign a non-competition agreement

LEGALESE: The Founders will be able to compete with the Company after their employment has ended.

PLAIN ENGLISH: He has 30 days to respond to the subpoena. Otherwise, he will be arrested and put into jail

LEGALESE: The Company will pay for reasonable attorneys’ fees and costs incurred by the Founders in responding to any subpoenas or other legal process, provided that the Company shall not be liable for any settlement payments made without its prior written consent.

PLAIN ENGLISH: If the company folds, the Series A investors will be the first to get their money back

LEGALESE: The Company will repay the Series A Investors before any other creditor in the event of its insolvency.

PLAIN ENGLISH: If a majority of the investors decide to sell their shares, the investors who disagree to sell their shares will be forced to do so.

LEGALESE: In the event of a sale or transfer by the Investors, all other Investors shall have preemptive rights to purchase such Investor’s interest on terms no less favourable to the Company than those offered to any third party.

These results are impressive given that GPT-3 was not trained specifically for the type of translation.

APPLICATION: verbal to formal written text translation

Another translation discourse example is in converting everyday verbal communication, filled with “ums” and “ahs” and general filler, to more compact and formal English. This may help in your business to more compactly represent the intent of voice recordings or casual communications between you and your customers. We form our prompt based on several examples of Oral:Written pairs as follows:

Oral: So yeah like how is he doing today?
Written: How is he doing today?

Oral: And so like alright let's tell him we should stick to the plan you know.
Written: Let's tell him to stick to the plan.

Oral: And I don't know it feels like things are going well.
Written: Things are going well.

Oral: Well uh I mean give me an example so
Written: Give me an example

Oral: That is a nice hat you are wearing.
Written: That is a nice hat you are wearing.

Technically, these pairs form a base prompt that guide GPT-3 into understanding the type of completions we are seeking given a new oral expression. Now that our base prompt is set, we simply tack on a final Oral:Written pair to complete the prompt but with the text of the Written completion omitted. This blank final Written entry in our prompt cues GPT-3 to complete it in the way that the other examples in the prompt were completed, i.e., form a Written counterpart to our Oral text. Here are several examples of Oral to Written translations performed by GPT-3:

Oral: Uh so when are we going to like meet with him?
Written: When are we going to meet with him?
Oral: um so like what day is he thinking?
Written: What day is he thinking?

Oral: Like dude you can't be like um serious.
Written: Like dude you can't be serious.
Oral: Is this something we you know have to like figure out or what?
Written: Is this something we have to figure out or what?
Oral: I told him doing a rad backflip off the cliff was like a bad idea man but I mean he didn't listen.
Written: I told him that doing a rad backflip off the cliff was a bad idea, but he didn't listen.

Impressive translations such as these can be used in many aspects of your business to directly help clients understand text in a particular language or style of communicating.

summary

Generative text models have received a lot of attention over the last year. Their ability to generate human-like text coupled with data trained on literally the entire world's text data on the internet has established a new modeling paradigm and likely will change our expectations of NLP forevermore.

While we wait to see how such model's evolve over time, in this article we presented one practical application of such models by using GPT-3 to bootstrap a text model by simulating themed text data. This process bypasses the standard means of gathering ground truth data, which oftentimes involves scraping web sites or patiently pinging APIs to data that is controlled or collected by a second party, e.g., Twitter or Reddit, saving time and money in getting your models and the ability to validate them off the ground.

We also explored tapping directly into the output of GPT-3 for the purposes of transforming text from one style to another, illustrating examples in transforming plain text into legalese and casual oral communication into more formal written form.

We hope that these examples serve to inspire you to explore and create generative text models such as GPT-3 to better your own business applications. While we focused on the use of GPT-3, it is a commercial product and you must get permission to use it from Open AI. A free alternative with no such restrictions is with AleutherAI's GPT-Neo and GPT-J.